Papers with controlled generation
Generating Text from Language Models (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a growing percentage of natural language processing tasks focus on the generation of text from probabilistic language models. |
| Approach: | They will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
| Outcome: | This tutorial will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on controlled text generation has required attribute-based fine-tuning of the base language model or restricted the parameterization of the attribute discriminator. |
| Approach: | They propose a global score-based alternative for controllable text generation that combines arbitrary pre-trained black-box models for achieving desired attributes in the generated text. |
| Outcome: | The proposed method outperforms methods that require extra training or fine-tuning . the proposed method is based on a model with energy values of a linear combination of scores from black-box models . |
DiffusPoll: Conditional Text Diffusion Model for Poll Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media platforms manipulate public opinion through sheer numbers and cause biases, authors say . they say new paradigm for poll generation can generate high-quality samples while preserving diversity . |
| Approach: | They propose a non-autoregressive diffusion model that uses masks to generate polls . they use attribute tags to enhance the quality of polls and to diversify poll options . |
| Outcome: | The proposed model matches the Transformer model while offering greater diversity and quality. |
Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks (2024.acl-long)
Copied to clipboard
| Challenge: | Disentangled latent spaces usually have better semantic separability and geometrical properties, which leads to better interpretability and controllable data generation. |
| Approach: | They propose a flow-based invertible neural network mechanism integrated with a transformer-based language Autoencoder to deliver latent spaces with better semantic separability and geometrical properties. |
| Outcome: | The proposed model can deliver latent spaces with better separability properties compared to the current state-of-the-art models. |
Sentence Bottleneck Autoencoders from Transformer Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining a language model on text have been used for building models in NLP, but they do not work for sentence representations derived from pretrainer models based on tokens or basic pooling operations. |
| Approach: | They propose to build a sentence-level autoencoder from a pretrained transformer language model. |
| Outcome: | The proposed model achieves better quality than previous methods on text similarity and style transfer tasks while using fewer parameters than large pretrained models. |
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2023.eacl-main)
Copied to clipboard
| Challenge: | Large-scale vision-language pre-trained (VLP) models generate unfaithful or nonsensical texts given the source input, which is called hallucination. |
| Approach: | They propose a VLP loss-based model to mitigate object hallucination by decoupling VLP objectives and a token-level image-text alignment. |
| Outcome: | The proposed model reduces object hallucination by 17.4% on two benchmarks. |
An Invariant Learning Characterization of Controlled Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Controlled generation is a problem of creating text that contains stylistic or semantic attributes of interest. |
| Approach: | They propose a distribution shift-based control system that can be used to train a predictor of the desired attribute. |
| Outcome: | The proposed method shows that the most effective predictor should be invariant across multiple text environments. |
Multilingual Summarization with Factual Consistency Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models generate factually inconsistent summaries, reducing their utility for real-world applications. |
| Approach: | They propose to use data filtering and controlled generation to detect hallucinations in machine generated summaries. |
| Outcome: | The proposed models detect factual inconsistencies in machine generated summaries, but they focus on English only. |
Controlled Generation with Prompt Insertion for Natural Language Explanations in Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies present tokens, examples, and hints for corrections, but do not directly explain the reasons in natural language. |
| Approach: | They propose a method called controlled generation with Prompt Insertion that uses Large Language Models to explain the reasons for corrections in natural language. |
| Outcome: | The proposed method can explain the reasons for corrections in natural language by guiding the LLMs to generate explanations for all correction points. |
Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior work explored the domain of controlled style generation, a task in which a generative language model aims to generate text with a specified style 2 . however in practice, text often contains not only a single style, but a combination of styles. |
| Approach: | They propose to use calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes to combine multiple styles in a reward function. |
| Outcome: | The proposed dynamic weighting outperforms static weighting approaches with respect style control while maintaining linguistic quality. |
SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
Text Embeddings Reveal (Almost) As Much As Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a vector database of dense text embeddings stores only the text data, not the original text . a multi-step method that iteratively corrects and re-embeds text can recover 92% of 32-token text inputs exactly. |
| Approach: | They propose a method that iteratively corrects and re-embeds text to recover 92% of 32-token text inputs exactly. |
| Outcome: | The proposed method recovers 92% of 32-token text inputs exactly. |
Women, Infamous, and Exotic Beings: A Comparative Study of Honorific Usages in Wikipedia and LLMs for Bengali and Hindi (2025.emnlp-main)
Copied to clipboard
| Challenge: | Honorifics encode nuanced socio-pragmatic cues such as power, age, gender, fame, and cultural distance. |
| Approach: | They propose to study third-person honorific usage across 10,000 Hindi and Bengali Wikipedia articles . honorifics are more prevalent in Bengali than in Hindi, while non-honorifics dominate . |
| Outcome: | The authors show that large language models internalize similar socio-pragmatic norms . their analysis shows that honorifics are more prevalent in Bengali than in Hindi . |